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ABSTRACT 


Is offered the set of criteria for assessing efficiency of the process forming the answers to 
multiple-choice test items. To increase accuracy of computer-assisted testing results, it is 
suggested to assess dynamics of the process of forming the final answer using the 
following factors: loss of time factor and correct choice factor. The model application 
results show the high efficiency of suggested set of criteria. 
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INTRODUCTION 

In web-oriented e-learning computer-assisted testing systems existing today, when 
assessing degree of conformity of the test item answer to the reference answer, final 
answer of the user is taken into account, but the answer formation process' dynamics 
(Rybanov, A.A., 2006a), by which it is possible to judge about (Bereby-Meyer, Y., Meyer, 
J., Flascher, O.M., 2002) choice by the user of the right answer at random, doubt of the 
user about correctness of his/her knowledge, use of prompts by the user, etc., is not 
taken into account. Ana lysis of the process of forming by the user the final answer is 
considered by example of multiple-choice test items of the type "several of several" 
(Figure: 1). 
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Figure: 1 

Example of the process of forming the answer to multiple-choice test item 
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In web-oriented e-learning systems, the final assessment 6 for the answer to the test 
item includes, as a rule, only comparison of the user's final answer (Rybanov, A.A., 
2006b) (Figure: 1, step 4) with the reference answer (Figure: 2) and is calculated 
according to the formula: 

e = P-5, (i) 

where (3 (0; 1 ] - is the test item's difficulty measure; 5- is degree of conformity of the 

test item answer to the reference answer. 
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Figure: 2 

Reference answer to the multiple-choice test item 

The same final answer to the test item can be obtained by the user with various 
trajectories of the process of the final answer formation, therefore the formation process 
dynamics should be taken into account when making final assessment for the test item 
(Rybanov, A.A., 2008). Response time is an additional source of information necessary for 
assessing abilities of the tested user, and also for analysing the testing process (Van Der 
Linden, W.J., 2008). 

To increase accuracy of computer-assisted testing results, it is suggested to assess 
dynamics of the process of forming the final answer using the following factors: 

> loss of time factor; 

> H - correct choice factor. 

The factor p takes into account loss of time, during forming final answer to test item, 
caused by choosing and subsequent cancelling the answer elements. For example, at step 
2 (Figure: 1), the user chooses the answer element "Relation", which is cancelled at step 
4 (Figure: 1). 

The factor r| characterizes correctness of choice by the user of the final answer, i.e. takes 
into account sequence of choosing correct and wrong answer elements taking into 
account their weight factors. 

For example, trajectory of the process of the final answer formation presented on Figure: 
1 {step 1: choosing "Attribute", step 2: choosing "Relation", step 3\ choosing 
"Relationship", step 4: cancelling "Relation") leads to the final answer {"Attribute", " 
Relationship"} and should have higher value of the criterion n than the trajectory of 
obtaining the same final answer: step 1: choosing "Relation", 
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step 2: choosing "Attribute", step 3: choosing "Relationship", step 4\ cancelling 
"Relation". I.e., the more quickly the user chooses correct answer elements with the 
biggest weight factors, the higher the correct choice factor n is. 

Thus, the final assessment 6 for the answer to the test item should be determined not 
only by the difficulty measure (5 and the degree of conformity of the test item answer to 
the reference answer 5, but also by the factors p and n . 

MATHEMATICAL FORMULATION 

The multiple-choice test item model of the type "several of several" can be presented as: 

T =(Q,E, r,0,p), (2) 

where Q - is the question contents; E = (e, \ i = 1, n) - is a set of elements for forming the 
answer to the test item; r = (y, | y, = Y, G (0;1), / = 1, n) - is a set of weight factors, y,- 
is weight factor of element e, ; D = (o', \ d, = /^ (<?,) , d, g B = {0,1}, / = l,n) - is the reference 
answer to the test item, d, - is descriptor of element e, of the answer (o', =1 if e, is 
element of the reference answer, otherwise o', = 0); |3 - is the test item's difficulty 
measure. 

For the sets D and r, the following conditions should be met: 

i < Id, < 2(i -d,), 2y rd, = 2y/-(i -d,) = l. (3) 

i=i i =i i=i /=i 

Example: 1. The multiple-choice test item on Figure: 1 can be formally presented as 

7- 1 = «?,£,r,0,|3), 

where Q, E, r, D, (3 are shown on Figure: 3. 
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The user's final answer to the test item T can be presented as a set: 

W = (w, | w-, SB = {0,1},/ =U7). (4) 

Degree of conformity of the test item answer to the reference answer can be presented 
as: 

n 

5 = f(w v w 2 . w n )= Iw r w r (2d, -1), (5) 

i=i 

where w, e B - is descriptor of / -th element of the final answer to the test item; 
f(w lt w 2 ,...,w n ) - is a function for assessing conformity of the user's final answer to the 
reference answer. 

Example: 2. The user's final answer to the multiple-choice test item T 1 : 

W = (1,0,1,0,1,1,0), 5 = f (1,0,1,0,1,1,0) = 0.70 . 

Let us name position in space W(tj) = ( w 1 (t J ),w 2 (t J ) . w n (tj )) gB" of the user's answer 

at fixed t, as image of the user's answer to the test item, and let us name the time f, as 
point of fixing the answer image W(tj). 

When forming the final answer, the image of the user's answer to the test item changes, 
drawing in a phase plane B" some curve, which can be named as trajectory of process of 
forming the answer to the test item. 

Parameters of conformity of the image of the user's answer I V(tj) to the reference 
answer D can be calculated according to the following formulas: 

a(tj)= Sy rd-w^tj), h(tj) = TiViil-d/YWitfj), (6) 

/=i /=i 

where a{tj ) is the sum of weights of correct answer elements marked in the image W(tj ) 
as correct elements; h(t y ) is the sum of weights of wrong answer elements marked in the 
image W[tj) as correct elements. 

Starting point of trajectory of process of forming the answer to the test item is the 
answer image W(t 0 ), which meets the following conditions: 

W(t Q ) =(w i (t Q ) | w, (f 0 ) = 0,/ = 1, n ). (7) 

Intermediate point of trajectory of process of forming the answer to the test item is the 
answer image W(t k ) , where t k is the time when the user sets (or unsets) the correct 
element mark (the action of "setting/unsetting the mark ') in the process of forming the 
answer. 

Final point of trajectory of process of forming the answer to the test item is the answer 
image W(t m ), where t m is the time when the user enters the final answer. 

Trajectory P of process of forming the answer to the multiple-choice test item can be 
presented as ordered sequence of the answer images: 

P r =(W(tj)\tj ^V r ,j =07m), (8) 
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( 9 ) 

( 10 ) 


which meets the following conditions: 

n _ 

H w i (t k _ 1 )-w i (t k )\=l, k = l,m-l, 

i =1 

W(tJ=W(t m _ 1 ), 

where V r = (tj \ j = 0, m) - is a set of points of fixing the answer image. 


The same answer to the multiple-choice test item can be obtained with various 
trajectories of the answer formation process. Example: 3. On Figure: 4 examples of 
trajectories of the process of forming the same final answer to the multiple-choice test 
item T 1 are presented. 
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Figure: 4 

Trajectories of the process of forming the answer to multiple-choice test item 

& 

Figure: 5 

Dynamic parameters a(t) and h(t) for trajectory P 2 
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Figure: 6 

Dynamic parameters a(t) and h(t) for trajectory P 5 
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On Figures. 5-6, dynamic characteristics a(t) and h(t) for trajectories A, and P 5 
(Example: 3) of the process of forming the final answer to multiple-choice test item T x 
are presented. 

Trajectory of the process of forming the answer reflects dynamics of choosing the answer 
to the test item. Taking into account that the user first chooses the answer elements, 
which he/she considers correct (factor n ), and also quantity of steps to the final answer 
(factor |j), the factor E, of general effectiveness of the process of forming the final answer 
can be used as a metric, which describes dynamics of the process of forming the answer. 

^ = nii ■ (ii) 

Taking into account the factor E, , the final assessment 6' for the tested user's answer to 
the test item will be calculated as follows: 

0' = (3-^5. (12) 

The factor r|of the correct choice in the process of forming the final answer to the test 
item can be presented as: 

m-1 

n = 7^r^ -# 

hgiV + bfaWM-t,) 

/=0 


(13) 

where g(t), b(t) are the functions of changing the sum of weights of all correct (wrong) 
answer elements chosen by the user in the process of forming the final answer to the test 
item. 

The values g(t ), b[t) show the sum of weights of all correct (wrong) answer elements at 
all points of fixing, starting from t 0 and including t ; . 

For multiple-choice test items of the type "several of several": 

g(tj)= Za(t k ), b(tj) = Eh(t k ). 

k =0 k =0 


(14) 


The loss of time factor p can be presented as: 


M = 


m 

lj(t,Ht,-t,. 1 ) 

_ [=1 _ 

m r 

/=1 
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where I (t), 0(t) are the functions of changing the quantity of the actions 




"setting/unsetting the mark' performed by the user in the process of forming the final 
answer to the test item. 

The values /(f y ), 0{tj ) show the quantity of the actions "setting/unsetting the mark' 
the user performed at all points of fixing, including f y , from the beginning t 0 of the 
process of forming the answer. 

Initial condition of the process of forming the final answer to the multiple-choice test 
item: / (t 0 ) = O(t 0 ) =0. 


For multiple-choice test items: 

ntj)= -w i (t J _ 1 )yw i (t J ), 

k=li=l 


( 16 ) 

0(tj)= 

k=l i =1 


(17) 

The factor l, of general effectiveness of the process of forming the final answer changes 
within the limits [0;1]. 

RESULTS AND DISCUSSION 

Let us consider trajectories P 1 , P,, P 3 , P_ (Example: 3) of forming by the user the same 
final answers to the test item T 1 . Let us order the trajectories P, ( r = 1,4 ) by the time t p 
of performing by the user the action "setting the mark' for the answer element e 5 with 
descriptor d 5 = 0 : t p = 35 < t p = 45 < t p _ = 55 < = 65 . Correct choice factors r\ r for 

trajectories P r form the similar sequence by index r: 
n 3 = 0.551 < n 4 = 0.720 < r) 2 = 0.870 < r)i = 0.950 that proves influence of sequence of user's 
choice of correct and wrong answer elements on factor r|. 

Let us consider trajectory P 5 (Example: 3), which describes process of forming the final 
answer during which the user performs the actions "setting the mark' for the answer 
elements e 1 (at t = 25 ) and e 4 (at t = 45) and then performs the actions "unsetting the 
mark' for e Y (at t = 55 ) and e 4 (at t = 10). On Figures: 7-8, dynamic characteristics g(t ), 
b(t) and / (t ), 0(t) for trajectory P 5 of the process of forming the final answer are 
presented. 
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Figure: 7 

Dynamic parameters g[t) and b(t) for trajectory P 5 
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Figure: 8 

Dynamic parameters /(f) and 0(t) for trajectory P 5 

Value of the loss of time factor p 5 for trajectory P 5 is less than value 14 for trajectory P 1 , 
during whose forming the user does not perform the action "unsetting the mark'. The 
result |_i 5 = 0.870 < p x = 1 proves dependence between the factor p and the action 
"unsetting the mark'. In Table: 1, the criteria, which describe dynamics of the process of 
forming by the user the answer to the test item for the Example: 3 trajectories are 
specified. 
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Taking into account dynamics of the process of forming the answer to the test item 


Trajectory of 
the process of 
forming the 
answer to the 
test item 
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.720 

.820 

.413 

Ps 

.700 

.820 

.691 

.870 

.601 

.820 

.345 


It follows from the Table: 1 that without taking into account dynamics of the process of 
forming the answer by the user, the final assessment 6 for the same answer to the test 
item is identical for various trajectories of the answer formation process. 


Taking into account dynamics of the process of forming the answer to the test item 
allows to make more objective assessment 6 ' = ( 3 -^ 5 . 

CONCLUSION 

The suggested set of criteria (11-17)-. r], p, 5, 0' which takes into account dynamics 
of the process of forming by user the answers to multiple-choice test items of the type 
"several of several", is introduced in LMS Moodle that ensures correct assessment of 
knowledge in distance education testing. 
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